076 GitHub copilot cost optimization - #1237
Conversation
Updated README to clarify prerequisites and challenges.
- Updated Challenge-05 to use GitHub Spec Kit with real commands - Added proper installation steps using uv tool install - Included full workflow: constitution → spec → plan → tasks → implement - Updated Coach guide with troubleshooting for uv installation - Added reflection questions and learning resources
- Add Part 1: Build app with ad-hoc prompting (no structure) - Add Part 2: Build same app with Spec Kit workflow - Add Part 3: Compare results between approaches - Remove installation instructions (moved to Challenge 0 prerequisites) - Clarify that Spec Kit is one of many spec-driven approaches - Update Coach guide to match new structure
feat(challenge-05): Replace spec-first with Spec Kit hands-on workflow
Adds Part 3 (judge quality on an open-ended planning task) after the car wash example, with a scoring rubric and cost tie-in. Renumbers the compare and classify sections to Parts 4 and 5. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…e students and changed the coach guidance accordingly
Removes the deterministic car wash task and promotes the open-ended pickleball planning task into Part 2. Rewords the intro, no-retry rule, compare/classify sections, success criteria, and reflection questions to fit a checkable + open-ended task pair instead of easy/hard. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Updated the challenge description to clarify the difference between direct prompting and spec-driven development using GitHub Copilot. Adjusted the structure and content to enhance understanding of the experiment objectives and requirements.
Removed additional challenge section regarding status field updates and comparisons.
Updated the Coach's Guide for Challenge 01 to enhance clarity and structure, including detailed instructions for comparing direct prompting with Spec-Driven Development.
There was a problem hiding this comment.
Pull request overview
Initial content drop for the new 076 – GitHub Copilot Cost Optimization What The Hack. It adds the student challenge materials, supporting resource assets (including a small NYC starter app), and coach guides/solutions (including a model-measurement harness).
Changes:
- Added the full student challenge write-ups (Challenges 00–06) and the hack landing README.
- Added student “Resources” content, including the CityScout NYC mini-app and Challenge 03 model-selection worksheets.
- Added coach guides and solution assets (including measured results + a reproducible harness).
Reviewed changes
Copilot reviewed 50 out of 53 changed files in this pull request and generated 9 comments.
Show a summary per file
| File | Description |
|---|---|
xxx-GitHubCopilotCostOptimization/Student/Resources/.devcontainer/devcontainer.json |
Duplicate/placeholder Resources devcontainer under xxx- path. |
076-GitHubCopilotCostOptimization/Student/Resources/Challenge06/data/sales.csv |
Challenge 06 sample dataset. |
076-GitHubCopilotCostOptimization/Student/Resources/Challenge-03-ModelSelection/README.md |
Student-facing instructions for Challenge 03 resource pack. |
076-GitHubCopilotCostOptimization/Student/Resources/Challenge-03-ModelSelection/prompts.md |
Verbatim prompts + answer key guidance for model comparison. |
076-GitHubCopilotCostOptimization/Student/Resources/Challenge-03-ModelSelection/output-constraints.md |
Output-constraint cheat sheet to reduce output tokens. |
076-GitHubCopilotCostOptimization/Student/Resources/Challenge-03-ModelSelection/measurements-template.md |
Tables/template for recording token usage and outcomes. |
076-GitHubCopilotCostOptimization/Student/Resources/Challenge-02-NYCApp/src/routes/restaurants.ts |
Restaurants route for the NYC demo app. |
076-GitHubCopilotCostOptimization/Student/Resources/Challenge-02-NYCApp/src/routes/neighborhoods.ts |
Neighborhoods route for the NYC demo app. |
076-GitHubCopilotCostOptimization/Student/Resources/Challenge-02-NYCApp/src/routes/events.ts |
Events route stub for students to modify in Challenge 02. |
076-GitHubCopilotCostOptimization/Student/Resources/Challenge-02-NYCApp/src/data/restaurants.ts |
Sample restaurants data. |
076-GitHubCopilotCostOptimization/Student/Resources/Challenge-02-NYCApp/src/data/neighborhoods.ts |
Sample neighborhoods data. |
076-GitHubCopilotCostOptimization/Student/Resources/Challenge-02-NYCApp/src/data/events.ts |
Sample events data. |
076-GitHubCopilotCostOptimization/Student/Resources/Challenge-02-NYCApp/README.md |
Student README for the NYC demo app. |
076-GitHubCopilotCostOptimization/Student/Resources/Challenge-02-NYCApp/.github/copilot-instructions.md |
Intentionally bloated Copilot instructions for scoping exercise. |
076-GitHubCopilotCostOptimization/Student/Resources/.gitkeep |
Keeps Resources folder in source control. |
076-GitHubCopilotCostOptimization/Student/Resources/.devcontainer/devcontainer.json |
Resources-local devcontainer configuration. |
076-GitHubCopilotCostOptimization/Student/Challenge-06.md |
Student instructions for the Token Golf competition. |
076-GitHubCopilotCostOptimization/Student/Challenge-05.md |
Student instructions for context window management. |
076-GitHubCopilotCostOptimization/Student/Challenge-04.md |
Student instructions for tool overhead + cache invalidation exercise. |
076-GitHubCopilotCostOptimization/Student/Challenge-03.md |
Student instructions for model selection + output constraints. |
076-GitHubCopilotCostOptimization/Student/Challenge-02.md |
Student instructions for context engineering + NYC app task. |
076-GitHubCopilotCostOptimization/Student/Challenge-01.md |
Student instructions for direct prompt vs spec-driven development. |
076-GitHubCopilotCostOptimization/Student/Challenge-00.md |
Student prerequisites/baseline setup instructions. |
076-GitHubCopilotCostOptimization/README.md |
Hack landing page + challenge index. |
076-GitHubCopilotCostOptimization/Coach/Solutions/Challenge-03-ModelSelection-Solutions/TODO.md |
Author TODOs / deferred items for Challenge 03 assets. |
076-GitHubCopilotCostOptimization/Coach/Solutions/Challenge-03-ModelSelection-Solutions/results.md |
Measured answer-key results for Challenge 03. |
076-GitHubCopilotCostOptimization/Coach/Solutions/Challenge-03-ModelSelection-Solutions/README.md |
Coach README for Challenge 03 harness + results. |
076-GitHubCopilotCostOptimization/Coach/Solutions/Challenge-03-ModelSelection-Solutions/harness.py |
Repro harness to measure pass rate + token usage via GitHub Models API. |
076-GitHubCopilotCostOptimization/Coach/Solutions/Challenge-02-NYCApp-Solutions/src/routes/restaurants.ts |
Coach solution for restaurants route. |
076-GitHubCopilotCostOptimization/Coach/Solutions/Challenge-02-NYCApp-Solutions/src/routes/neighborhoods.ts |
Coach solution for neighborhoods route. |
076-GitHubCopilotCostOptimization/Coach/Solutions/Challenge-02-NYCApp-Solutions/src/routes/events.ts |
Coach solution for /events/free-this-week. |
076-GitHubCopilotCostOptimization/Coach/Solutions/Challenge-02-NYCApp-Solutions/src/data/restaurants.ts |
Coach solution data: restaurants. |
076-GitHubCopilotCostOptimization/Coach/Solutions/Challenge-02-NYCApp-Solutions/src/data/neighborhoods.ts |
Coach solution data: neighborhoods. |
076-GitHubCopilotCostOptimization/Coach/Solutions/Challenge-02-NYCApp-Solutions/src/data/events.ts |
Coach solution data: events. |
076-GitHubCopilotCostOptimization/Coach/Solutions/Challenge-02-NYCApp-Solutions/README.md |
Coach README describing the optimized Challenge 02 solution. |
076-GitHubCopilotCostOptimization/Coach/Solutions/Challenge-02-NYCApp-Solutions/.github/skills/free-events-endpoint/SKILL.md |
Example coach skill for the free-events endpoint pattern. |
076-GitHubCopilotCostOptimization/Coach/Solutions/Challenge-02-NYCApp-Solutions/.github/instructions/events.instructions.md |
Example scoped instruction file for events route work. |
076-GitHubCopilotCostOptimization/Coach/Solutions/Challenge-02-NYCApp-Solutions/.github/instructions/api.instructions.md |
Example scoped instruction file for API routes. |
076-GitHubCopilotCostOptimization/Coach/Solutions/Challenge-02-NYCApp-Solutions/.github/copilot-instructions.md |
Example reduced global Copilot instructions. |
076-GitHubCopilotCostOptimization/Coach/Solutions/.gitkeep |
Keeps Solutions folder in source control. |
076-GitHubCopilotCostOptimization/Coach/Solution-06.md |
Coach guide for Token Golf competition. |
076-GitHubCopilotCostOptimization/Coach/Solution-05.md |
Coach guide for context window management challenge. |
076-GitHubCopilotCostOptimization/Coach/Solution-04.md |
Coach guide for MCP tools overhead + cache invalidation challenge. |
076-GitHubCopilotCostOptimization/Coach/Solution-03.md |
Coach guide for model selection + output constraints challenge. |
076-GitHubCopilotCostOptimization/Coach/Solution-02.md |
Coach guide for context engineering challenge. |
076-GitHubCopilotCostOptimization/Coach/Solution-01.md |
Coach guide for direct prompt vs SDD challenge. |
076-GitHubCopilotCostOptimization/Coach/Solution-00.md |
Coach guide for prerequisites/baseline challenge. |
076-GitHubCopilotCostOptimization/Coach/README.md |
Coach landing page + guide index. |
076-GitHubCopilotCostOptimization/.wordlist.txt |
Wordlist for spelling/lint tooling. |
.devcontainer/xxx-GitHubCopilotCostOptimization/devcontainer.json |
Duplicate/placeholder devcontainer under xxx- path. |
.devcontainer/devcontainer.json |
Existing repo-level devcontainer definition (dotnet) present in repo. |
.devcontainer/076-GitHubCopilotCostOptimization/devcontainer.json |
Devcontainer for the 076 hack workspace. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| - ❌ unrelated layers (frontend, security, logging, testing) | ||
|
|
||
| 👉 Students should REDUCE this drastically. | ||
| `` No newline at end of file |
| eventsRouter.get('/free-this-week', (req, res) => { | ||
| return res.json({ | ||
| count: events.length, | ||
| events | ||
| }); | ||
| }); |
|
|
||
| ## Description | ||
|
|
||
| All teams will receive the same coding task with defined acceptance criteria from your coach. The task specification (found in the `/Challenge06/` folder of Resources.zip) includes functional requirements, tests that must pass, required documentation, and code quality standards. |
|
|
||
| **You must complete the Token Golf task yourself before the hack:** | ||
|
|
||
| 1. Implement the coding task specified in `/Student/Resources/Challenge06/token-golf-task.md` |
| "codespaces": { | ||
| "openFiles": [ | ||
| "076-GitHubCopilotCostOptimization/Student/Resources/README.md" | ||
| ] | ||
| } |
| "codespaces": { | ||
| "openFiles": [ | ||
| "README.md" | ||
| ] | ||
| } |
| { | ||
| "name": "076-GitHubCopilotCostOptimization", | ||
| "image": "mcr.microsoft.com/devcontainers/typescript-node:1-22-bookworm", | ||
| "hostRequirements": { | ||
| "cpus": 2 |
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 50 out of 53 changed files in this pull request and generated 13 comments.
Suppressed comments (21)
Previously missed (13) — in code that hasn't changed since the last review.
076-GitHubCopilotCostOptimization/Student/Challenge-02.md:30
- Agent Skills are discovered and loaded on demand when Copilot determines they are relevant; they do not require explicit invocation. This instruction teaches the wrong activation model and may invalidate the context-cost comparison. Describe the skill as relevance-triggered/on-demand instead.
076-GitHubCopilotCostOptimization/Student/Challenge-02.md:118 - This path-specific instruction guidance contradicts the supported structure already described on line 28. Scoped instructions belong under
.github/instructions/and useapplyTofrontmatter; placing/api/.github/inside the source tree will not create the intended scope.
076-GitHubCopilotCostOptimization/Student/Challenge-03.md:34 - This tells students to rerun anomalous results immediately after the first-shot rule forbids reruns. That changes the sample count and undermines a consistent comparison. Keep the measurement single-shot here; discuss repeated validation separately from the recorded run.
076-GitHubCopilotCostOptimization/Coach/Solution-02.md:98 - Copilot does not scope repository instructions by walking nested
.github/copilot-instructions.mdfiles. The reference solution correctly uses.github/instructions/*.instructions.mdwithapplyTofrontmatter, so this troubleshooting advice sends coaches toward an unsupported structure.
GitHub Copilot walks up the directory tree looking for `.github/copilot-instructions.md` files. Ensure:
- Directory structure is correct
- File is named exactly `.github/copilot-instructions.md` (not `.github/copilot-instructions-api.md` in the api folder)
076-GitHubCopilotCostOptimization/Coach/Solution-06.md:26
- All five challenge references are shifted: instruction scoping is Challenge 02, model selection is 03, session/tool hygiene is 04, compaction is 05, and spec-first development is 01. The incorrect mapping will make coaches point participants to the wrong prior material.
This issue also appears on line 40 of the same file.
- Global instructions scoped appropriately (Challenge 01)
- Efficient model selection (Challenge 02)
- Clean session hygiene (Challenge 03)
- Proactive compaction (Challenge 04)
- Spec-first approach (Challenge 05)
076-GitHubCopilotCostOptimization/Coach/Solutions/Challenge-02-NYCApp-Solutions/.github/instructions/events.instructions.md:11
- This scoped rule requires
location, but neither the student nor coach event fixtures contain that field, and the stated endpoint acceptance criteria require only name/date/price behavior. The completed solution therefore violates its own instructions; align the required fields with the provided schema.
- Ensure each event has name, date, and location
076-GitHubCopilotCostOptimization/Coach/Solutions/Challenge-03-ModelSelection-Solutions/harness.py:116
- The coach README says the harness defaults to
n=4, and all published reference measurements use four trials, but the actual default is 50. Running the documented command without-ntherefore incurs 12.5× as many API calls and can heavily rate-limit premium models.
ap.add_argument("-n", type=int, default=50)
076-GitHubCopilotCostOptimization/Coach/Solution-02.md:21
- Skills are selected automatically when their metadata matches the current task, not only when explicitly invoked;
@workspace /newis also unrelated to loading the added Agent Skill. This coach explanation would reinforce the same incorrect activation model taught to students.
2. **GitHub Copilot Skills (conditional cost):**
- Only load when explicitly invoked
- Example: `@workspace /new` only loads creation templates when called
.devcontainer/devcontainer.json:8
- This newly added default devcontainer is for hack 031 and opens
/workspace/031-DevOpsWithGitHub, not the 076 materials introduced by this PR. Because it occupies the default.devcontainer/devcontainer.jsonlocation, users can launch an unrelated .NET 6 environment instead of the 076 TypeScript/Python configuration. Remove this file or replace it with the 076 configuration.
"name" : "DevOps with GitHub WTH in Codespaces",
// Or use a Dockerfile or Docker Compose file. More info: https://containers.dev/guide/dockerfile
"image": "mcr.microsoft.com/devcontainers/dotnet:0-6.0",
"workspaceFolder": "/workspace/031-DevOpsWithGitHub",
"workspaceMount": "source=${localWorkspaceFolder},target=/workspace,type=bind,consistency=cached",
076-GitHubCopilotCostOptimization/README.md:25
- Challenge 01 explicitly excludes automated tests and frames the exercise as an observation, not proof that SDD reduces token burn. This summary instead promises a deterministic-control result the exercise neither implements nor requires. Align the overview with the actual direct-prompt-versus-spec comparison.
- Challenge 01: **[Spec-Driven Development](Student/Challenge-01.md)**
- Demonstrate that deterministic controls (tests, specs) reduce trial-and-error token burn
076-GitHubCopilotCostOptimization/Coach/Solution-01.md:1
- This guide does not use Spec Kit: it deliberately uses a single manually created
sdd.mdwith no Spec Kit commands, constitution, plan, or tasks. Calling it “with Spec Kit” contradicts the student exercise and the rest of this coach guide.
# Challenge 01 - Spec-Driven Development with Spec Kit - Coach's Guide
076-GitHubCopilotCostOptimization/Coach/Solution-05.md:15
- These exact context-utilization thresholds are presented as deterministic behavior, yet the guide provides no measurement basis and later requires degradation after 70%. Quality degradation varies by model, task, and context composition, so a participant may run the experiment correctly and still fail the stated criterion. Label these as hypotheses/illustrative ranges and grade the recorded observation rather than a required outcome.
- 0-60% capacity: Quality remains high
- 60-70% capacity: First signs of degradation (inconsistencies)
- 70-85% capacity: Noticeable quality loss, forgotten requirements
- 85-100% capacity: Severe degradation, expensive recovery loops
076-GitHubCopilotCostOptimization/Coach/README.md:105
- These bullets contradict the explicit statement on lines 65–67 that the hack requires no Azure resources. Remove this leftover template content so coaches are not told to prepare nonexistent Azure dependencies.
This issue also appears on line 107 of the same file.
- Azure resources that will be consumed by a student implementing the hack's challenges
- Azure permissions required by a student to complete the hack's challenges.
076-GitHubCopilotCostOptimization/Student/Challenge-06.md:17
Student/Resources/Challenge06contains onlydata/sales.csv; the task specification and tests promised here are missing. Consequently, teams cannot know the acceptance criteria, run the required tests, or complete the capstone. Addtoken-golf-task.mdand all referenced test/starter files to the resource folder.
076-GitHubCopilotCostOptimization/Coach/Solution-06.md:44- The same off-by-one challenge mapping is repeated in the required scorecard, so teams would attribute every optimization technique to the wrong challenge.
- Which instruction scoping strategy (Challenge 01)
- Output constraints used (Challenge 02)
- Tools pruned (Challenge 03)
- Compaction points (Challenge 04)
- Spec-first approach (Challenge 05)
076-GitHubCopilotCostOptimization/Coach/Solutions/Challenge-02-NYCApp-Solutions/src/routes/events.ts:19
- This comparator also subtracts
Dateobjects, which TypeScript rejects. Compare their numeric timestamps instead.
.sort((a, b) => new Date(a.date) - new Date(b.date));
076-GitHubCopilotCostOptimization/Student/Resources/.devcontainer/devcontainer.json:21
- This resource-level configuration also opens a nonexistent
Student/Resources/README.md, so no guide appears when the container starts. Point it to an existing resource README.
.devcontainer/076-GitHubCopilotCostOptimization/devcontainer.json:23 - The configured startup file does not exist:
Student/Resourceshas noREADME.md. Codespaces will fail to open the intended guide. Point this to an existing resource entry page, such as the CityScout README.
"076-GitHubCopilotCostOptimization/Student/Resources/README.md"
076-GitHubCopilotCostOptimization/Coach/Solution-06.md:14
- The documented Token Golf task does not exist in
Student/Resources/Challenge06, so the coach cannot perform this mandatory pre-work or establish par. Add the complete task/test assets before making this a critical prerequisite.
**You must complete the Token Golf task yourself before the hack:**
1. Implement the coding task specified in `/Student/Resources/Challenge06/token-golf-task.md`
2. Use optimization techniques from Challenges 1-5
076-GitHubCopilotCostOptimization/Student/Resources/Challenge-02-NYCApp/.github/copilot-instructions.md:147
- This stray two-backtick marker is unmatched and renders as unexplained literal content in the instruction file. Remove it.
076-GitHubCopilotCostOptimization/Coach/README.md:109 - A complete “Suggested Hack Agenda” and “Repository Contents” already appear above; this second section is untouched template guidance with sample days and duplicated headings. Remove the duplicate template block (lines 107–131) before publishing the coach guide.
## Suggested Hack Agenda (Optional)
_This section is optional. You may wish to provide an estimate of how long each challenge should take for an average squad of students to complete and/or a proposal of how many challenges a coach should structure each session for a multi-session hack event. For example:_
| 1. Locate the `baseline-task.md` file in the `/Challenge00/` folder of the Resources.zip | ||
| 2. Complete the coding task described using GitHub Copilot Chat in your normal working style (don't optimize yet!) |
| Give **both** models this exact prompt: | ||
|
|
||
| ``` | ||
| I want to build a pickleball app. Produce an implementation plan. | ||
| First, ask me any clarifying questions you need. Then identify the |
| - Use `/context` to see which tools are enabled and how much of your token budget they consume | ||
| - The VS Code Output panel (View → Output → "GitHub Copilot") shows actual tool invocations | ||
| - Each MCP tool's schema (name, description, parameters) is sent to the model on every request | ||
| - System prompt changes (like editing instruction files or toggling tools) can invalidate cached context | ||
| - Use `/usage` to see your credit breakdown after each task |
| Using the NYC App you worked with in Challenge 02 (in `Resources/Challenge-02-NYCApp`), implement these four features in order: | ||
|
|
||
| - **Feature 1 – Restaurant Filtering:** Add query-parameter support to `GET /restaurants` so callers can filter by `borough` and/or `cuisine` (e.g., `/restaurants?borough=Brooklyn&cuisine=Mexican`). Return the full list when no filters are provided. | ||
| - **Feature 2 – Neighborhood Detail with Restaurants:** Add `GET /neighborhoods/:name` that returns the matching neighborhood object plus an embedded array of restaurants in that neighborhood (a cross-file join between `neighborhoods.ts` and `restaurants.ts` data). |
| - **Feature 3 – Pagination (cross-cutting):** Add `?page=N&limit=N` support to all three list endpoints (`/events/free-this-week`, `/restaurants`, `/neighborhoods`). Default to page 1, limit 10. Return `{ page, limit, total, data }`. | ||
| - **Feature 4 – Input Validation & Error Handling:** Add proper `400 Bad Request` responses for invalid query parameters and `404 Not Found` for missing resources across all routes. | ||
|
|
||
| These four features touch multiple files, require cross-file references, and create natural breakpoints between each feature—ideal for testing context management strategies. We recommend not committing these files to make it easier to revert. |
| return False | ||
| if wi == -1: | ||
| return True | ||
| return di < wi |
| Using the NYC App you worked with in Challenge 02 (in `Resources/Challenge-02-NYCApp`), implement these four features in order: | ||
|
|
||
| - **Feature 1 – Restaurant Filtering:** Add query-parameter support to `GET /restaurants` so callers can filter by `borough` and/or `cuisine` (e.g., `/restaurants?borough=Brooklyn&cuisine=Mexican`). Return the full list when no filters are provided. |
| - `/compact` to summarize and continue in the same session | ||
| - `/clear` to wipe and restart with a fresh prompt | ||
| - Start a new session (`/new`) at any breakpoint you choose |
| ] | ||
| } | ||
| }, | ||
| "postCreateCommand": "curl -LsSf https://astral.sh/uv/install.sh | sh && ~/.local/bin/uv tool install specify-cli --from git+https://github.com/github/spec-kit.git", |
| ] | ||
| } | ||
| }, | ||
| "postCreateCommand": "curl -LsSf https://astral.sh/uv/install.sh | sh && ~/.local/bin/uv tool install specify-cli --from git+https://github.com/github/spec-kit.git", |
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
This is the initial pull request for the new GitHub Copilot Cost Optimization WTH